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Docket No.: K0600.0208/P208 
IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 



In re Patent Application of: 
Shiro Fujieda 

Serial No. : Not Yet Assigned Group Art Unit: Not Yet Assigned 

Filed: December 6, 1999 Examiner: Not Yet Assigned 

For: IMAGE PROCESSING 

APPARATUS AND METHOD, 
MEDIUM STORING PROGRAM 
FOR IMAGE PROCESSING, AND 
INSPECTION APPARATUS 



Assistant Commissioner for Patents 
Washington, D.C. 20231 

FIRST PRELIMINA RY AMENDMENT 

Dear Sir: 

Prior to examination on the merits, please amend the above-identified U.S. 
patent application as follows: 

In the Claims: 

Claim 3, lines 1 and 2, please delete "or 2". 



Claim 5, lines 1 and 2, please change "any one of claims 1 to 4", to 



Claim 6, lines 1 and 2, please change "any one of claims 1 to 4", to 



Claim 7, lines 1 and 2, please change "any one of claims 1 to 6", to 



--claim 1— . 
--claim 1—. 
--claim 1--. 



Serial No.: Not Yet Assigned ' Docket No.: K0600.0208/P208 

Claim 8, lines 1 and 2, please change "any one of claims 1 to 5", to —claim 1— . 

Claim 9, lines 1 and 2, please change "any one of claims 1 to 6", to --claim 1—. 

Claim 10, lines 1 and 2, please change "any one of claims 1 to 9", to -claim 1— 

Claim 11, lines 1 and 2, please change "any one of claims 1 to 9", to —claim 1 — 

Claim 12, lines 1 and 2, please change "any one of claims 1 to 11", to 
-claim 1—. 

Claim 13, lines 1 and 2, please change "any one of claims 1 to 11", to 
—claim 1--. 

Claim 16, lines 1 and 2, please change "any one of claims 1 to 15", to 
—claim 1— . 

Claim 17, lines 1 and 2, please change "any one of claims 1 to 15", to 
—claim 1—. 

Claim 19, lines 1 and 2, please change "any one of claims 1 to 18", to 
--claim 1—. 

2 

1 084595 v1, N8VN011.DOC 



►Serial No.: Not Yet Assigned ' Docket No.: K0600.0208/P208 

Claim 20, lines 1 and 2, please change "any one of claims 1 to 15 and claim 



19", to --claim 1— . 



Claim 26, lines 1 and 2, please change "any one of claims 23 to 25", to 



--claim 23—. 



Claim 41, line 1, please delete "or 38". 



Claim 42, lines 1 and 2, please change "any one of claims 37 to 41", to 



--claim 37—. 



REMARKS 



Claims 3, 5, 6-13, 16, 17, 19, 20, 26, 41 and 42 are amended to delete the 
multiple dependency. No new matter has been entered. 

In view of the above, each of the presently pending claims in this application is 
believed to be in immediate condition for allowance. Accordingly, the Examiner is 
respectfully requested to pass this application to issue. 

Dated: December 6, 1999 Respectfully submitted, 




Registration No.: 28,371 
DICKSTEIN SHAPIRO MORIN & 
OSHINSKY LLP 
2101 L Street NW 
Washington, DC 20037-1526 
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DESCRIPTION 



IMAGE PROCESSING APPARATUS AND METHOD, 



MEDIUM STORING PROGRAM FOR IMAGE PROCESSING, 



5 



AND INSPECTION APPARATUS 



Technical Field 

The present invention relates generally to an apparatus 
for and a method of processing a gray level image, a medium 
10 having a program for image processing stored thereon, and an 
apparatus for inspecting the presence or absence, the 
position, the shape, the direction, etc. of an object. 
Background Art 



15 subjected to recognition processing on a gray level image, a 
method of binarizing the gray level image representing a 
recognition object at a predetermined threshold has been 
conventionally known. The recognition processing based on 
binary data is easily affected by illumination variations, 

20 shading and so forth. Particularly when an X-ray 

transmitting image is processed, as in the inspection of a 
void in a soldered portion on a substrate, a lot of noises 
are produced on the image, so that a contrast between the 
object and a background is decreased. Accordingly, the 

25 image may, in some cases, be unrecognizable. 

A method of subjecting a gray level image to normalized 
correlation operation processing using a model image of an 



When an object having a given contour shape is 
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object, to recognize the position of the object has been 
known. In recognition processing using a normalized 
correlation operation, the model image of the object must be 
previously set. Therefore, the method is unsuitable for a 
5 case where the size of the object varies. Further, the 

normalized correlation operation must be executed in each of 
scanning positions on the image, thereby increasing a 
hardware configuration and processing time. Particularly in 
order to measure the direction of the object, it is 

10 necessary to rotate the model image a predetermined angle at 
a time and execute the normalized correlation operation for 
each rotation angle position, thereby increasingly 
complicating a device configuration and processing. 

In order to measure the position and the direction of 

15 an object in a contour shape having corners, for example, a 
chip component on a substrate, a method of tracking an edge 
on an image to specify straight lines corresponding to the 
contours of the object, then finding an equation of each of 
the straight lines, and further recognizing the position of 

20 a point of intersection of the plurality of straight lines 
found by the equations as the position of the component may, 
in some cases, be used. However, this method complicates 
processing such as the tracking of the edge and the 
operation of the equations. 

25 With respect to an object having boundary lines whose 

shapes are point-symmetrical or an object like a polygon at 
least two sets of opposite sides of which are parallel, a 
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method of measuring the position and the shape of the object 
using the direction of each of edges appearing on an image 
has been proposed (see JP-A-9-229646 ) . 

According to this method, a representative point of the 
5 object can be extracted with high precision without being 
affected by illumination variations and complicating a 
device configuration. In this method, however, a 
recognition object is limited to a polygon having boundary 
lines whose shapes are point-symmetrical or a polygon at 
10 least two sets of opposite sides of which are parallel. 

Accordingly, it is impossible to recognize an object whose 
contours are asymmetrical. A representative point finally 
found is a point in the contours of the object. 
Accordingly, it is difficult to apply this method to 
15 processing for extracting the corners of an object that is a 
polygon and measuring the direction of the object. 
Disclosure of Invention 

An object of the present invention is to make it 
possible to stably process even a low-contrast image and an 
20 image including noises. 

Another object of the present invention is to make it 
unnecessary to set a model image and make an operation 
performed by an operator simple. 

Still another object of the present invention is to 
25 make a configuration and processing simple. 

A further object of the present invention is to make it 
possible to make restrictions on an object as few as 
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possible. 

An image processing apparatus according to the present 
invention comprises gradient calculation means for 
calculating at least the direction of the level gradient of 
5 a processing unit in given image data including a plurality 
of pixels, the pixels respectively having level data; line 
segment formation means for producing line segment image 
data representing a line segment having a direction 
corresponding to the direction of the level gradient which 

10 is calculated by the gradient calculation means and a given 
length; and line segment image storage means for storing the 
line segment image data produced by the line segment 
formation means. 

Data representing the level of each of the pixels is 

15 gray level data represented by two or more bits . 

Although the processing unit is basically a pixel, it 
may be a block composed of a plurality of pixels. 

The data representing the level of the pixel represents 
brightness (darkness) or density if it is luminance data. 

20 Consequently, the level gradient can be also represented 
using terms such as "brightness gradient", "darkness 
gradient", and "density gradient". The image data is not 
limited to the luminance data. For example, it may be data 
representing a color (for example, red, green, or blue). In 

25 this case, the level gradient will represent the change in 
the density of the color. In either case, the direction of 
the level gradient is the direction of the maximum gradient 
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(the magnitude of the level gradient takes a positive or 
negative maximum value) . 

A direction corresponding to the direction of the level 
gradient includes various modes depending on the purposes of 
5 image processing. The first is the direction in which the 
level increases (a positive direction), and the second is 
the direction in which the level decreases (a negative 
direction). The third is a direction perpendicular to the 
direction of the level gradient. The direction 

10 perpendicular to the direction of the level gradient 

includes two directions. It may be either one or both of 
the directions. 

As an example, there is an image (an image of an 
object) representing a circle (or a shape close thereto), as 

15 shown in Fig. la. It is assumed that the inside of the 

circle is relatively bright, and a background is relatively 
dark. A level gradient that is not zero is calculated in 
the position (a pixel) of the contour C of the circle or its 
vicinity. (For convenience of drawing, it should be 

20 understood that although the contour C is clearly drawn by a 
broken line, the level (brightness, darkness, or density) of 
image data in the vicinity of the contour C is generally 
changed more rapidly, as compared with those in the other 
portions. It is assumed that the change in the level takes 

25 the maximum value in a direction perpendicular to the 

direction of the tangent of the contour C. (The same is 
true for the other example, described later. ) It is assumed 
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that the brighter the image is, the higher the level of the 
image data is (actually, the reverse is, of course, 
possible). When it is assumed that a line segment having a 
predetermined length (described later) is formed in the 
5 direction in which the level increases (a positive 

direction) from the pixel at which the level gradient which 
is not zero has been calculated, a lot of line segments L 
which are directed to the center of the circle from the 
contour C are drawn, as shown in Fig. lb. Consequently, the 

10 position of the contour C or the presence of the circle 

becomes clear. If the line segment L is made longer, the 
center of the circle is found. Contrary to this, when it is 
assumed that a line segment having a suitable length is 
formed in the direction in which the level decreases (a 

15 negative direction) from the pixel at which the level 

gradient which is not zero has been calculated, a lot of 
line segments L directed toward the outside of the circle 
radially from the contour C are drawn, as shown in Fig. lc. 
The position of the contour C or the presence of the circle 

20 becomes clear. 

As another example, there is an image (an image of an 
object) representing a rectangle, as shown in Fig. 2a. It 
is assumed that the inside of the rectangle is relatively 
bright, and a background is relatively dark. A level 

25 gradient that is not zero is calculated at a pixel 

representing the contour C of the rectangle. The direction 
of the level gradient that is not zero is a direction 
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perpendicular to the contour C. When it is assumed that 
line segments L having a predetermined length are formed 
from the pixel at which the level gradient that is not zero 
has been calculated in two directions perpendicular to the 
5 direction of the level gradient, the line segment L which is 
overlapped with the contour C and the line segment L 
extending onto an extension of the contour C are drawn, as 
shown in Fig. 2b. The presence of the rectangle, the 
position of the contour C, the direction of the rectangle 

10 and so forth become clear. 

Image processing, for example, detection or recognition 
of an object image, described later (detection of the 
presence, detection of the position, or detection of the 
direction) or display of a line segment image is performed 

15 on the basis of the line segment image data stored in the 
line segment image storage means. 

According to the present invention, the level gradient 
of image data is calculated for each processing unit (for 
example, for each pixel) on a given image, and a line 

20 segment image representing a line segment having a direction 
corresponding to the calculated direction of the level 
gradient and a given length is obtained. Even if the 
contour of an image is not necessarily clear, a clear line 
segment is obtained. Consequently, even a low-contrast 

25 image and an image including noises can be subjected to 
stable image processing. The configuration or the 
processing is also relatively simple. A basic pattern (a 
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model image) need not be set. 

In one embodiment of the present invention, there is 
provided image storage means for storing the given image 
data. The gradient calculation means calculates a level 
5 gradient for each processing unit with respect to the image 
data stored in the image storage means. 

The image storage means does not necessarily need to 
temporarily store all image data. For example, it is also 
possible to calculate a level gradient in real time while 
10 shifting image data along a plurality of scanning lines 
which are adjacent to one another on a plurality of line 
memories . 

The above-mentioned given image data may be image data 
of one frame which is picked up by a video camera (or a 

15 television camera) (including a still camera), image data 

corresponding to one screen which is stored in a memory, or 
image data in a part region extracted from the image data 
corresponding to one screen. The region will be generally 
defined by setting a window. 

20 In one embodiment of the present invention, there is 

further provided image data extraction means for extracting 
image data in a processing region set in input image data 
and feeding the extracted image data to the gradient 
calculation means. 

25 The input image data includes image data obtained by 

picking up an object using a video camera, image data stored 
in a memory, image data transmitted from another apparatus, 



and so forth. 

In a preferred embodiment of the present invention, 
there is further provided means for setting the processing 
region. 

5 The setting means includes various modes. The first is 

a mode of displaying an image represented by input image 
data on a display device, and designating a portion to be a 
processing object (a region including an object image) using 
a cursor or the like on a display screen of the display 

10 device. The setting means will include a display device and 
an input device for providing input for moving the cursor. 

The second is a mode of setting a region of a size 
previously determined (or inputted) at a predetermined 
position. In this case, the setting means will include 

15 storage means for storing the position and the size and 

means for determining a processing region on image data on 
the basis of the storage means. 

A line segment formed by the line segment formation 
means may be represented by image data at a gray level (two 

20 or more bits) or image data at a binary level. When the 

line segment is represented at the gray level, the level may 
be fixed or variable. In the case of the binary level, line 
segment image data takes a value which is either "1" or "0". 
The line segment image data may be expressed by data 

25 representing the initial point (coordinates), the terminal 
point (coordinates), and the level (value) of the line 
segment. 



In a preferred embodiment, the gradient calculation 
means calculates the magnitude of the level gradient in 
addition to the direction of the level gradient. In this 
case, it is preferable that the line segment formation means 
5 produces line segment image data having a level (brightness, 
darkness, or density (including the density of a color) 
corresponding to the calculated magnitude of the level 
gradient. Consequently, a line segment is drawn at the 
level corresponding to the magnitude of the level gradient. 
10 In a more preferred embodiment, the magnitude of the 

level gradient which is calculated by the gradient 
calculation means and a predetermined threshold level are 
compared with each other, to produce a line segment image 
only when the magnitude of the level gradient is not less 
15 than the predetermined threshold level (the level of the 

line segment image may be fixed or a value corresponding to 
the magnitude of the level gradient, as described above). 
Since the level gradient has a positive or negative value, 
as described above, it should be herein understood that it 
20 is judged whether or not the absolute value of the level 
gradient is not less than the predetermined threshold (a 
positive value). 

When the calculated level gradient is low, no line 
segment image is produced to avoid affection of small level 
25 changes and noises. The predetermined threshold may be 
determined from the point of view of whether or not the 
small level changes and the noises are excluded from a 
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processing object as disturbances. 

In Fig. lb, the length of the line segment is set to a 
small value. When the length of the line segment is larger 
than the radius of the circle, a lot of line segments are 
5 drawn in an overlapped state at the center and its vicinity, 
as shown in Fig. Id. (In Figs. Id and 3b, the number of 
line segments is decreased for easy understanding.) 

There are two methods of storing line segment image 
data in line segment image storage means for storing images 

10 of a lot of line segments. 

The first is a method of adding new line segment image 
data (level data) to line segment image data (level data) 
(including zero) already stored at each of the pixels and 
storing the result of the addition. The line segment image 

15 storage means shall comprise addition means . This method is 
applicable to both of a case where the line segment image 
data is represented at a gray level and a case where it is 
represented at a binary level. The addition includes 
processing for converting new line segment image data (for 

20 example, multiplying the line segment image data by a 

coefficient) and adding the converted line segment image 
data. With respect to the pixel at which the line segments 
are drawn in an overlapped state, the level of the image 
data is increased (accumulated) every time the line segment 

25 is overlapped with the others. 

The second is a method of storing new line segment 
image data without subjecting the line segment image data to 
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addition processing. As a result, with respect to the pixel 
at which image data (excluding zero) representing a line 
segment image has already been stored, the stored data is 
maintained as it is (the new line segment image data may be 
5 overwritten or may not be written). Only with respect to 
the pixel (which is zero) at which no image data is stored, 
new line segment image data for the pixel is newly written. 
Although the method is also applicable to both of a case 
where the line segment image data is represented at a gray 

10 level and a case where it is represented at a binary level, 
it is particularly effective for the case where it is 
represented at a binary level. For example, when the line 
segment image data is represented by a binary number "1" , 
image data which is represented by "1" are concentrated in a 

15 portion where line segments are concentrated. 

There are some modes as to the length of a line segment 
to be formed and the positions of the initial point and the 
terminal point of the line segment when a gradient level 
which is not zero is calculated. 

20 The first is a mode of producing a line segment having 

a given length in a direction corresponding to the 
calculated direction of a level gradient from the position 
of a processing unit (the center of a pixel or a block), as 
described with reference to Figs, lb to Id. In this case, 

25 the length of the line segment may be previously set or 
inputted . 

The second is a mode of giving the distance d from the 
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position of a processing unit (the center of a pixel or a 
block) e to the initial point of a line segment L and the 
distance D from the position e of the processing unit to the 
terminal point of the line segment L. In this case, the 
5 line segment formation means calculates the positions of the 
initial point and the terminal point using the given 
distances d and D and the calculated direction of a level 
gradient, and produces the line segment L from the 
calculated initial point to the calculated terminal point in 

10 a direction corresponding to the calculated direction of the 
level gradient. 

Fig. 3b illustrates an example of a line segment L 
drawn when a value which is smaller than the radius of a 
circle and is not zero is used as the distance d and a value 

15 which is larger than the radius of the circle and is smaller 
than the diameter thereof is used as the distance D. Fig. 
3c illustrates an example in which both the distances d and 
D are set to a value equal to the radius of the circle. A 
line segment L to be produced is a point (one pixel) (the 

20 point shall be included in the line segment). Fig. 3d 

illustrates an example in which d = 0 and D = <» (infinity) 
and a line segment is drawn outside a circle. A processing 
region W containing the circle is set, and the boundary of 
the processing region W is the position where the distance 

25 equals infinity. In other words, a line segment having a 
length of infinity is drawn from the position of a 
processing unit. 
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The length of the line segment, the distances d and D, 
etc. may be determined depending on the purposes of 
processing. 

In a preferred embodiment, there is further provided 
5 means for setting or inputting the length of the line 
segment, the distances d and D, etc. A user may only 
perform an operation for setting or inputting the values. 
Accordingly, the operation is easy. The values may, of 
course, be previously set depending on the purpose of 

10 processing. 

In order to detect the presence or absence of an object 
image in a particular shape or of a size (a shape or a size 
in an allowable range) and detect the direction or the 
position of its center, for example, the image processing 

15 apparatus comprises, in a preferred embodiment, means for 
detecting a portion where line segments represented by the 
line segment image data stored in the line segment image 
storage means are concentrated. The detection means is 
applied to both of a case where the line segment image data 

20 is written into the line segment image storage means upon 
being added and a case where it is written without being 
added. However, it is particularly effective for the line 
segment image storage means for storing the line segment 
image data represented at a binary value without adding the 

25 line segment image data. 

As an example, in a region or an area of a suitable 
size (for example, 10 pixels by 10 pixels), when the number 
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of pixels at which a line segment exists (a pixel at which a 
line segment is drawn or a pixel having data representing a 
line segment) is large (when it exceeds a predetermined 
threshold), it can be said that the region is a portion 
5 where the line segments are concentrated. The region is 
scanned over a processing range (region). 

In another preferred embodiment, the image processing 
apparatus comprises means for detecting the position of the 
pixel having the maximum of the levels of the line segment 

10 image data stored in the line segment image storage means. 
The detection means is effective for the line segment image 
storage means for storing the line segment image data while 
adding the line segment image data. 

It is preferable that the image processing apparatus 

15 further comprises means for judging whether or not the 
maximum level of the result of addition (the result of 
accumulation) of the line segment image data exceeds a 
predetermined threshold. 

Detection of the presence (position) of the portion 

20 where the line segments are concentrated, detection of the 

position of the pixel having the maximum level, or detection 
of the presence (position) of the maximum level exceeding a 
predetermined threshold by the detection means is applied to 
various types of inspections. The threshold will be 

25 determined by trial-and-error depending on the purpose of 
the inspection. 

In the example shown in Fig. 3b, for example, the 
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presence and the position of a circle having a certain 
radius (or a radius having a value close thereto) or a shape 
close thereto (an object having such a shape) are detected. 
In a case where a circle having a radius from a to b (a < b) 
5 or a shape close thereto is detected, a radius a may be 

employed as the distance d from a processing unit (a pixel, 
etc.) to the initial point of a line segment (d = a), and a 
radius b may be employed as the distance D from the 
processing unit (the pixel, etc.) to the terminal point of 

10 the line segment (D = b) . So long as the radius of a 

circular shape (an object) is in a range from a to b, line 
segments are concentrated in the vicinity of its center, or 
the maximum level exceeds the threshold. Accordingly, it is 
judged that a circle whose center is a portion where the 

15 line segments are concentrated (or a shape close thereto) 
exists. Similarly, it is possible to detect a shape or an 
object having a circular arc-shaped contour. 

In the example shown in Fig. 3c, when there exists a 
circle having a radius rofr=d=D (or a shape close 

20 thereto), line segments (points) are concentrated at its 
center, or the maximum level exceeds a threshold. 
Accordingly, it is possible to confirm the presence of such 
a circle and the position of its center. The example is 
applicable to not only the inspection of the position of the 

25 center but also the inspection of the radius and the 
inspection of out of roundness. 

Consequently, image data representing an image of an 
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inspection object can be used for inspecting the presence 
and the position of an object having a particular shape, the 
presence and the position of a defective region appearing as 
a change in the level of the image data, and the like. 
5 According to the image processing apparatus, the 

position of the circular shape or the object is not 
necessarily specified sufficiently accurately- However, at 
least the presence and the rough position of a predetermined 
circular shape or object become clear. If an accurate 

10 position and shape must be specified, therefore, another 
method such as an edge extracting method may be used. 

Still another application is the inspection of a 
chamfer radius (a radius of curvature) of an object whose 
corners are chamfered in a round shape. In this case, the 

15 minimum value and the maximum value of a tolerance for the 
chamfer radius may be respectively taken as d and D. Only 
when the chamfer radius is within the tolerance, therefore, 
line segments are concentrated, or the maximum level exceeds 
a predetermined threshold. 

20 In the example shown in Fig. 2b, in a contour and an 

angle of a rectangle, line segments are concentrated, or the 
maximum level exceeds a threshold. Consequently, it is 
possible to detect a linear contour of the rectangle or an 
object, the presence and the position of a point of 

25 intersection of two or more straight lines which is 
represented by the angle or corner, the direction of 
arrangement, and so forth. This is applicable to the 
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inspection of the size, the inspection of the area, the 
inspection of the position of the center of gravity, the 
inspection of the direction (the direction of a principal 
axis, the direction of a side, or the direction of a 
5 diagonal), etc. 

It is also possible to compare the level of image data 
of each of pixels stored in the line segment image storage 
means storing accumulated line segment image data with a 
predetermined threshold, and form (draw) a shape represented 

10 by the pixel having data whose level exceeds the threshold. 
The effect of disturbances such as noises can be eliminated. 

In various types of inspections, described above, an 
object is preferably imaged by a camera. Image data 
obtained by the imaging is fed to gradient calculation 

15 means . The image processing apparatus has imaging means 
such as a camera. 

In a still preferred embodiment, there is provided a 
display device for displaying a line segment image 
represented by the line segment image data representing a 

20 line segment formed by the line segment formation means or 

the line segment image data stored in the line segment image 
storage means. By seeing the display, it is possible to 
confirm or check the process of processing by the gradient 
calculation means, the line segment formation means or the 

25 line segment image storage means through vision. Only a 

part of a line segment having a level that is not less than 
a predetermined threshold may be displayed. 
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It is preferable that on a display screen of the 
display device, an image (an object image) picked up by the 
imaging means or an image (an object image) represented by 
image data which is a processing object by the gradient 
5 calculation means is displayed in addition to the line 

segment image with the image further overlapped therewith. 
Further, an edge of the object image may be extracted, and 
an image represented by the extracted edge may be displayed 
with the image overlapped with the line segment image. 

10 Desirably, it is possible to choose whether or not the 

object image is displayed and whether or not the edge image 
is displayed. It goes without saying that both the object 
image and the edge image may be displayed with the images 
overlapped with the line segment image. 

15 In a desirable embodiment of the present invention, a 

mark representing the portion, where the line segments are 
concentrated, the position of a pixel having the maximum 
level or the pixel having the maximum level exceeding a 
threshold, detected in the above-mentioned manner, is 

20 displayed with the mark overlapped with the object image or 
the edge image. The object image and the edge image may be 
displayed in an overlapped state. Switching can be also 
performed such that either one of the object image and the 
edge image is displayed. The line segment image may be 

25 displayed with the image further overlapped therewith. 

The present invention further provides an image 
processing apparatus comprising image processing means for 
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calculating at least the direction of the level gradient of 
a processing unit in given image data, and producing line 
segment image data representing a line segment having a 
direction corresponding to the calculated direction of the 
5 level gradient; and display means for displaying a line 
segment image represented by the line segment image data 
produced by the image processing means. 

The process of processing by the image processing means 
can be visually confirmed or checked through the line 

10 segment image displayed on the display device. 

The image (the object image) represented by the image 
data may be displayed with the image overlapped with the 
line segment image. Alternatively, it is preferable that an 
edge of the image represented by the image data is 

15 extracted, an image represented by the extracted edge is 
displayed in addition to or in place of the object image 
represented by the image data. It is also possible to 
provide means for switching the display of the object image 
and the display of the edge image. 

20 The present invention further provides a method of 

performing all the above-mentioned image processing, and a 
medium storing a program for controlling a computer so as to 
cause the computer to perform all the image processing. 

When the image processing apparatus according to the 

25 present invention is defined from another point of view, it 
is expressed as follows. That is, the image processing 
apparatus according to the present invention comprises means 
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for extracting a plurality of edges whose level gradients 
are not less than a predetermined value in given image data; 
means for setting, for each of the edges, a line segment 
extending in a direction corresponding to the direction of 
5 the extracted edge; and means for detecting the presence or 
absence of a point of intersection of a plurality of line 
segments and the position thereof. 

The direction of the line segment is typically a 
direction perpendicular to the direction of the edge or the 

10 same direction as the direction of the edge. 

It is possible to detect edges with respect to at least 
two points on the image, set line segments to judge the 
presence or absence of a point of intersection of the line 
segments, and calculate, when the point of intersection 

15 exists, the position of the point of intersection. The 

shape, the position, the direction, etc. of the object image 
can be judged by detecting one or a plurality of points of 
intersection. 

The present invention further provides an inspection 
20 apparatus suitable for various types of inspections, 
described above. 

The inspection apparatus comprises image input means 
for inputting image data representing a processing object; 
means for calculating at least the direction of the level 
25 gradient of a processing unit in the input image data, and 
producing line segment image data representing a line 
segment having a direction corresponding to the calculated 
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direction of the level gradient; and detecting the presence 
or absence of a portion where line segment images are 
concentrated or are overlapped with one another and the 
position thereof on the basis of the produced line segment 
5 image data. 

The direction corresponding to the direction of the 
level gradient is typically the direction of the level 
gradient or a direction perpendicular to the direction of 
the level gradient. It is preferable to produce, only with 

10 respect to the level gradient whose magnitude is not less 
than a predetermined value, the line segment. 

A portion where line segment images are concentrated is 
as described above. The portion where the line segment 
images are overlapped with one another means a pixel at a 

15 point of intersection of the line segments, a pixel at which 
an accumulated value of line segment image data exceeds a 
predetermined threshold, a pixel at which the accumulated 
value is the largest, and a pixel at which the accumulated 
value is the largest and exceeds the predetermined 

20 threshold. 

The inspection apparatus according to the present 
invention is simple in operation. For example, an operator 
need not set a reference pattern (a model image). Further, 
it is simple in configuration and processing. The shape, 

25 the size, etc. of an inspection object are hardly 

restricted. It is also applicable to an unclear image or an 
image which is low in contrast without being affected by 
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disturbances such as illumination variations and noises. 

In one embodiment, there is provided a display device 
for displaying the line segment image on the basis of the 
line segment image data. The process of producing the line 
5 segment image will visually become clear. 

It is preferable that an image of the object 
represented by the input image data is displayed on the 
display device with the image overlapped with the line 
segment image. 

10 In another embodiment, there is provided a display 

device for displaying the detected position of the portion 
where the line segment images are concentrated or are 
overlapped with one another on the image of the object 
represented by the input image data or an image represented 

15 by an edge extracted from the input image data. 

In order to adapt the inspection apparatus to the 
purposes of various types of inspections, there is further 
provided means for inputting data relating to the length of 
the line segment and the initial point and the terminal 

20 point of the line segment. 

An inspection apparatus according to the present 
invention can be utilized for the inspection of the presence 
or absence and the position of a defective region in an 
object, the inspection of the presence or absence and the 

25 position of a circle or a circular arc in the object, the 
detection of the center of the circle or the circular arc, 
the measurement of the radius, the inspection of out of 
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roundness, the inspection of the presence or absence and the 
position of a side or a corner of the object, and the 
inspection of the size, the area, the position of the center 
of gravity, the direction, etc. 
5 The other features or aspects of the present invention 

will become more apparent in the detailed description of the 
embodiments with reference to the accompanying drawings. 
Brief Description of Drawings 

Fig. la illustrates an example of an object image, and 
10 Figs, lb to Id illustrate examples of a line segment image 
produced with respect to the object image; 

Fi g. 2a illustrates another example of an object image, 
and Fig. 2b illustrates an example of a line segment image 
produced with respect to the object image; 
15 Rig. 3 a explains a method of drawing a line segment, 

and Figs. 3b to 3d illustrate examples of a produced line 
segment image; 

_Ei g. 4 i llustrates a configuration for producing an X- 
ray transmitting image supplied to an image processing 
20 apparatus; 

, Fig^ ^JS^ is a block diagram showing the electrical 
configuration of an , image processing apparatus according to 
one embodiment of the present invention; 

Fig. 6 illustrates an example of an input image; 
25 Fig . 7 illustrates an example of a local region for 

calculating a level gradient; 

Figs. 8a to 8h illustrate the relationship among a 
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target pixel, image data in its vicinity, and the direction 
of a level gradient; 

Fig. 9 illustrates an example of the directions of 
level gradients in an inspection area; 
5 Fig. 10 illustrates a line segment image produced in 

the inspection area; 

Figs. 11a to lid illustrate examples of a method of 
drawing line segments; 

F igs. 12a and 12b are flow charts showing the procedure 
10 for processing in an image processing apparatus; 

Figs. 1 2a_ and_13 b illustrate examples of display of the 
result of inspection; 

F±g_^_JA^ illustrates an example of a gray level image of 
an object having a cross contour shape; 
15 Fig. 15 illustrates an example of the directions of 

level gradients on the contours of an image with respect to 
the object shown in Fig. 14; 

Fig. 16 is a flow chart showing another example of the 
procedure for processing in an image processing apparatus; 
20 Fig. 17 illustrates the positions of points of 

intersection of line segments; and 

Fig. 18 is a block diagram showing another example of 
the electrical configuration of an image processing 
apparatus . 

25 Best Mode for Carrying Out the Invention 

An embodiment is directed to an apparatus for 
processing an X-ray transmitting image of a substrate on 
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which a package of LSI (Large Scale Integration) having 
terminals of solder balls arranged in an array on lower 
surface thereof is mounted. A molten solder (including a 
solder bump) shall be referred to as a solder ball in the 
5 present embodiment- The image processing apparatus inspects 
whether or not a void exists in the solder ball. 

Fig. 4 illustrates a configuration for producing image 
data fed to the image processing apparatus. 

A substrate 18 which is an inspection object is 
10 supported between an X-ray irradiation section 16 and an X- 
ray conversion section 17 by a supporting mechanism (not 
shown), and a camera 20 is disposed in a position below the 
X-ray conversion section 17. The X-ray irradiation section 
16 irradiates the substrate 18 with X-rays from its very 
15 small X-ray generation source. The X-rays which have passed 
through the substrate 18 are converted into visible light 
rays in the X-ray conversion section 17. The camera 20 
picks up the visible light rays obtained by the conversion, 
to produce image data representing an X-ray transmitting 
20 image representing the internal state of each of solder 
balls, and feeds the produced image data to the image 
processing apparatus. 

Fig. 5 illustrates the electrical configuration of the 
image processing apparatus. 
25 The image processing apparatus comprises an image input 

section 2, an image memory 3, an image output section 4, a 
line segment image memory 5, a character memory 6, a control 
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section 11, a timing control section 12, a monitor display 
device 13, an I/O port 14, and so forth. 

The image input section 2 comprises an analog-to- 
digital (A/D) converter for converting an analog image 
5 signal from the camera 20 into digital image data. Digital 
gray level image data produced in the image input section 2 
is stored in the image memory 3, and is preserved until the 
subsequent image data is inputted. 

The line segment image memory 5 stores line segment 
10 image data representing a line segment image, described 

later. The line segment image data in the memory 5 is used 
for purposes such as detection processing of a void and 
display of the line segment image on the monitor display 
device 13. When the luminance level of the line segment 
15 image, described later, is represented by eight bits, the 
memory 5 can store 16 bits or 32 bits per pixel. 

The character memory 6 stores data such as text data 
for displaying the result of inspection on the monitor 
display device 13 and data representing the position where 
20 the result of inspection is displayed. 

The memories 5 and 6 and the image memory 3 are 
connected to the control section 11 through an address/data 
bus, respectively, and outputs the stored data to the image 
output section 4 in response to an instruction from the 
25 control section 11 and a timing signal from the timing 
control section 12. 

The image output section 4 comprises an image synthesis 
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circuit for synthesizing output data from the memories 3, 5, 
and 6 and a digital-to-analog (D/A) conversion circuit, and 
produces an image signal for displaying the result of 
inspection by the circuits and outputs the produced image 
5 signal to the monitor display device 13. 

The control section 11 is mainly constituted by a CPU 
7, a ROM 8 and a RAM 9. The control section 11 further 
comprises a hard disk 10 in which a control program for 
causing the CPU 7 to execute a series of procedures for 

10 inspection has been installed. The CPU 7 accesses the 
memories 3, 5 and 6 through the address /data bus on the 
basis of the control program which has been installed in the 
hard disk 10, to perform processing for judging whether or 
not each of the solder balls on the substrate which is an 

15 inspection object is good (acceptable) and output the result 
of the processing to the monitor display device 13. 

The I/O port 14 is for connecting input sections such 
as a keyboard and a mouse and output sections such as an 
external storage device and a transmission section. 

20 Inspection conditions and inspection area setting data, 

described later, are inputted from the input sections. The 
final result of the inspection in a form of a file is 
outputted to the output sections. 

Fig. 6 illustrates an example of an image represented 

25 by gray level image data stored in the image memory 3. 
Images of a plurality of solder balls 21 appear more 
brightly than a background (an image of a substrate). On 
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the images of some of the solder balls 21, images of voids 
22 inside the solder balls appear. The image of the void 22 
is slightly darker than the image of the solder ball 21. 
The CPU 7 sets, for each of the solder balls , an 
5 inspection area r k (k = 1 to 6) of a size surrounding the 
solder ball in the input image shown in Fig. 6. In each of 
the inspection areas r k , the direction and the magnitude of 
the level gradient of the gray level image data are 
calculated for each pixel. The CPU 7 further produces a 

10 line segment having a direction along the calculated 

direction of the level gradient and a predetermined length 
(a line segment is produced only with respect to a case 
where the magnitude of the level gradient is not less than a 
predetermined value, as described later). An image of the 

15 line segment is also a gray level image, and the level 

thereof corresponds to the calculated magnitude of the level 
gradient. Line segment image data thus produced is stored 
in the line segment image memory 5 (the level thereof is 
accumulated for each pixel, as described later). The line 

20 segment image data may be image data having level data for 
each pixel, or can be composed of data representing the 
initial point, the terminal point and the level of the line 
segment. 

When the calculation of level gradients for all pixels 
25 in each of the inspection areas and the production of 

necessary line segment images are terminated, the CPU 7 
detects the degree of density of the line segment images or 
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the pixel having the maximum level, compares the maximum 
level with a predetermined threshold , and finally judges 
whether or not the image of the void 22 exists in the 
inspection area. 

5 Before describing the specific procedure for processing 

in the image processing apparatus , description is made of 
processing for producing a line segment image and the 
principle of judgment of the presence or absence of a void 
based on the line segment image. 

10 In processing for calculating a level gradient in image 

data, differential processing is performed, for each of 
pixels in an inspection area, in a local region (area) of a 
predetermined size which has the pixel as its center, to 
calculate level gradients along the X-axis and the Y-axis 

15 and finally find a level gradient at each of the pixels 
using the calculated level gradients. 

For example, a local region (3 pixels by 3 pixels) is 
set for a pixel g at a coordinate position (x, y) , as shown 
in Fig. 7. In differential processing by the Sober method, 

20 level gradients dx(x, y) and dy(x, y) along the respective 
axes are respectively calculated by equations (1) and (2) 
using a luminance (brightness) level I of each of the 
pixels : 

dx(x,y)={I(x+l,y-l)+2-I(x+l,y)+I(x+l,y+l)} 
25 -{I(x-l,y-l)+2-I(x-l,y)+I(x-l,y+l)} 

... Eq. (1) 

dy(x,y)={I(x-l,y+l)+2 -I(x,y+l)+I(x+l,y+l) } 
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-{I(x-l,y-l)+2-I(x,y-l)+I(x+l,y-l)} 

... Eg. (2) 

The level gradients along the X-axis and the Y-axis are 
respectively represented by vectors having magnitude 
5 dx(x,y)and magnitude dy(x,y) and directed toward the X-axis 
and the Y-axis, respectively. Consequently, a level 
gradient at the pixel g is represented by synthesizing the 
vectors toward the X-axis and the Y-axis. The magnitude 
EJXjy) of a composite vector is given by the following 
10 equation, and the direction thereof is a direction 
indicating the maximum gradient. 

Ei(x,y) =j(dx(x,y)) 2 + (dy(x,y)) 2 ... Eq. (3) 

Figs. 8a to 8h illustrate the direction of a level 

gradient depending on a luminance level distribution of a 
15 pixel g e and pixels in its vicinity by taking various 

examples. A hatched portion has a higher luminance value 

(is brighter) . 

A vector F is a composite vector of vectors 

respectively having level gradients along the X-axis and the 
20 Y-axis. In an example as illustrated, the vector F is 

directed from a high luminance level to a low luminance 

level. However, the direction of the vector F is determined 

depending on which direction is selected as the positive 

direction of the level gradient. 
25 The direction of a level gradient is represented by an 

angle E c (x, y) in a counterclockwise direction to the 

composite vector F from a vector B along the positive 
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direction of the X-axis starting from the pixel g e , assuming 
that the vector B is at an angle of zero degree. 

The vector F is set in any one of angle ranges 0 to 90 
degrees, 90 to 180 degrees, 180 to 270 degrees, and 270 to 
360 degrees with respect to the reference vector B depending 
on the magnitude of the level gradient along each of the X- 
axis and the Y-axis (positive, negative, or zero). The 
angle E c (x, y) indicating the direction of the level gradient 
is calculated by any one of the following equations (4) to 
(9) depending on conditions by the magnitudes of the level 
gradients dx(x, y) and dy(x, y) (a unit is degree). The 
magnitude of the level gradient at the pixel g e is calculated 
by the foregoing equation ( 3 ) because it corresponds to the 
length of the vector F. 
If dx(x,y)<0 and dy(x,y)>0, then 

E c (x,y)=-arctan(dx(x,y)/dy(x,y) ) — Eq. (4) 

If dx(x,y)>0 and dy(x,y)*,0, then 

E c (x,y)=180-arctan(dx(x,y)/dy(x,y) ) ... Eq. (5) 
If dx(x,y)<;0 and dy(x,y)<0, then 

E c (x,y)=360-arctan(dx(x / y)/dy(x,y) ) ... Eq. (6) 
If dx(x,y)>0 and dy(x,y)=0, then 

E c (x,y)=180 ... Eq. (7) 

If dx(x,y)<;0 and dy(x,y)=0, then 

E c (x,y)=0 ... Eq. (8) 

If dx(x,y)=0 and dy(x,y)>0, then 

E c (x,y)=90 ... Eq. (9) 

Fig. 9 illustrates only the contours of images in an 
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inspection area r 2 which are extracted from the input image 
shown in Fig. 6, where the contour of an image of a solder 
ball and the contour of an image of a void are illustrated. 
Vectors F x and F 2 respectively representing level gradients 
5 calculated with respect to representative points e x and e 2 on 
the contours are also illustrated. 

The image of the solder ball is higher in luminance 
level than a background, so that the vector F 1 directed from 
the point e 1 on the contour of the solder ball toward the 

10 background is obtained at the point e 1 . On the other hand, 
the image of the void is lower in luminance level than the 
image of the solder ball, so that the level gradient at the 
point e 2 on the contour of the void is represented by the 
vector F 2 directed inward from the position of the point e 2 . 

15 Fig. 10 illustrates a drawing of line segments 

respectively representing the directions of level gradients 
at all points (pixels) (in a range which can be illustrated) 
on the contour of the solder ball and the contour of the 
void which are illustrated in Fig. 9 (it is assumed that the 

20 levels of line segment images are constant). 

On the contour of the solder ball, all line segments L x 
are directed outward (toward the background). 

On the other hand, on the contour of the void, all line 
segments L 2 are directed toward the inside of the void. All 

25 the line segments L 2 cross one another or are overlapped with 
one another inside the void. The crossing or the 
overlapping of the line segments L 2 are important to judge 
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whether or not the image of the void exists inside the 
inspection area. In this sense, the length of the line 
segment or the positions of the initial point and the 
terminal point of the line segment are important. 
5 Figs. 9 and 10 are drawn on the assumption that the 

contours are clear. In an actual image, however, the 
boundary between portions of the image (a background, a 
solder ball, and a void) may not, in many cases, be clear. 
Accordingly, line segments are not directed in a perfect 

10 radial shape, and the initial point of the line segment does 
not necessarily exist on the circumference. However, the 
line segments drawn in the above-mentioned manner are clear, 
and the boundary clearly rises from its entire image. 

Figs. 11a to lid illustrate examples of line segments 

15 which differ in the length, the initial point and the 

terminal point, where line segments obtained as a result of 
taking representative points e p and e q on a circular contour 
and respectively finding level gradients for the points are 
illustrated. 

20 In Fig. 11a, two line segments L p and L q respectively 

start at pixels e p and e q which are level gradient 
calculation objects for producing the line segments, and 
have lengths larger than the diameter of the circle. When 
line segments are drawn for all points on a contour, the 

25 line segments are overlapped with one another, and an entire 
image represented by the line segments is larger than the 
circle (the contour). Although the presence of the circle 
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can be detected, the size and the center thereof, for 
example, are difficult to detect. 

In Fig. lib, the distance d from a pixel e q to the 
initial point of a line segment L q is larger than the radius 
5 of a circle, and the distance D from the pixel e q to the 
terminal point of the line segment L q is larger than the 
diameter of the circle {the same is true for a line segment 
L p ) . The line segments are concentrated in an annular shape 
around the contour of the circle. Although the presence of 

10 the circle can be also detected in this case, the center and 
the size thereof are difficult to detect. 

In Fig. 11c, line segments L p and L q respectively have 
pixels e p and e q as initial points, and the lengths thereof 
are smaller than the radius of a circle. Consequently, the 

15 line segments are concentrated along the inside of the 
circle, and do not exist at the center of the circle. 
Although the presence of the circle can be also detected in 
this case, the center thereof is difficult to detect. 

In Fig. lid, the distance d from a pixel e q to the 

20 initial point of a line segment L q is smaller than the radius 
of a circle, and the distance D from the pixel e q to the 
terminal point of the line segment L q is larger than the 
radius of the circle and smaller than the diameter of the 
circle. In this case, the line segments are concentrated in 

25 the vicinity of the center of the circle. With respect to a 
pixel at which the line segments are overlapped with one 
another, if the luminance levels of all the line segments 



35 



passing through the pixel are added (accumulated), since the 
largest number of line segments are overlapped with one 
another at the center of the circle, a value obtained by the 
addition is the largest at the center of the circle. The 
5 maximum level of the value obtained by the addition is 

detected, thereby making it possible to detect the center of 
the circle. This is the same as that previously described 
using Fig. 3b. 

Consequently, the positional relationship (the distance 
10 d) between a pixel and the initial point of a line segment 
and the length (the distance D) of the line segment must be 
set (determined) in advance depending on the size of a void 
which should be judged to be defective. 

It is possible to detect the void merely depending on 
15 whether or not there are line segments (the degree of 

density) without adding the luminance levels of the line 
segments passing through each of pixels. That is, it is 
judged that the higher the number of pixels at which the 
line segments are drawn in a local area (for example, 10 
20 pixels by 10 pixels) is, the more highly the line segments 
are concentrated. 

Such an idea can be utilized for applications of 
extracting only an object having a radius in a certain range 
when the object is approximately circular. For example, 
25 when it is desired to extract an object having a radius from 
a to b, values of a and b are entered from a keyboard. The 
image processing apparatus sets the distance from the 
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position of a pixel to the initial point of a line segment 
to a (d = a), and sets the distance from the position of the 
pixel to the terminal point of the line segment to b (D = 
b) . Consequently, concentration of line segments appears so 
5 long as the radius of the object is from a to b. If the 

density of the line segments can be detected, therefore, it 
can be judged that an object exists in the place where the 
line segments are concentrated. 

The distances d and D can be also set to the same 

10 value. In this case, the length of the line segment is 
zero. Actually, only one pixel is a line segment. 
Consequently, line segments (points) are concentrated in a 
very narrow region in the case where the object is a circle 
whose radius has a set value (d = D), while being dispersed 

15 in the other cases. Therefore, it is possible to know the 

degree of an error from a target radius of the circle or the 
degree of out of roundness from the degree of the 
dispersion. 

The distance D from the pixel to the terminal point of 
20 the line segment may be infinite. This actually means that 
a line segment is drawn until it reaches an end of a region 
represented by a line segment image memory. If d = 0 and D 
= oo are taken as default values, it is possible to detect 
that there exists a circle or a circular arc irrespective of 
25 the radius. Particularly as shown in Fig. 3d, it is 

effective for a case where a line segment is drawn toward 
the outside of a circle. 
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In the present embodiment, the distance D from the 
position of the pixel to the terminal point of the line 
segment and the distance d from the position of the pixel to 
the initial point of the line segment are set as inspection 
5 conditions on the basis of the above-mentioned principle, 
thereby detecting a void whose size is not less than a 
predetermined value as a detection object. The inspection 
can be also performed again by changing the inspection 
conditions as required. 

10 = Figs. 12a and 12b show the procedure for control of a 

frame of image inputted from the camera 20 in the image 
processing apparatus. The distance D from the position of a 
pixel to the terminal point of a line segment and the 
distance d from the position of the pixel to the initial 

15 point of the line segment which are to be inspection 

conditions shall be inputted prior to carrying out the 
procedure and stored in the RAM 9. 

When an image signal from the camera 2 0 is stored in 
the image memory 3 upon being digitally converted at the 

20 step STO, the CPU 7 displays the input image on the monitor 
display device 13 through the image output section 4. 

An operator designates an inspection area so as to 
include arbitrary one of solder balls on a display image 
using the mouse or the like. The CPU 7 recognizes the 

25 position where the inspection area is set (a positional 

relationship relative to the solder ball) and the size of 
the region on the basis of the designated position. It is 
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assumed that the position of each of the solder balls on a 
substrate (a position determined on the basis of the origin 
of coordinates on the substrate, for example) has been 
inputted previously or at this time point. The CPU 7 also 
5 sets, with respect to each of the solder balls other than 
that in the designated inspection area, the inspection area 
using the positional relationship between the solder ball 
and the inspection area (step ST1) (see Fig. 6). The 
position of the inspection area is represented by 

10 coordinates (x kl , y kl ) and (x k2 , y^) at upper left and lower 
right vertexes of the region. 

The CPU 7 proceeds to the step ST2 . At the step ST2 , 
level data representing each of pixels in the line segment 
image memory 5 is cleared to zero, and processing at the 

15 steps ST3 to ST19 is then repeatedly performed for each 
inspection area. 

At the step ST3, the initial position of a target pixel 
(a pixel as to which level gradient is to be calculated) g 
is determined to be the position (x kl , y kl ) of the upper left 

20 vertex of the inspection area. At the step ST4, level 

gradients dx(x, y) and dy(x, y) along the X-axis and the Y- 
axis at the target pixel g are respectively calculated on 
the basis of the equations (1) and (2). When both dx and dy 
are zero, the steps ST 6 to ST10 are skipped. If one of dx 

25 and dy is not zero, an angle E c (x, y) indicating the 

direction of the level gradient in accordance with any one, 
which is adapted to dx and dy, of the equations (4) to (9), 
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and the magnitude E ± {x, y) of the level gradient are 
calculated { step ST6 ) . 

It is then checked whether or not the calculated 
magnitude E i (x, y) of the level gradient is larger than a 
5 predetermined threshold E iLB ( step ST7 ) . If the magnitude 

E i( x f y) °f "t ne level gradient is not more than the threshold 
E^, the processing at the steps ST8 to ST10 of forming a 
line segment is not performed. A line segment is not formed 
with respect to a level gradient which is not an object to 

10 be detected, for example, nonunif ormity of luminance in an 

input image, thereby omitting useless processing, preventing 
an erroneous result of inspection from being obtained, and 
increasing the speed of processing as a whole. 

The CPU 7 determines the direction E c (x, y) as the 

15 direction of a line segment in a counterclockwise direction 
at a position (x, y) of the target pixel g. In the set 
direction, points which are respectively spaced apart from 
the target pixel g by the distances d and D are set as the 
initial point and the terminal point of a line segment 

20 corresponding to the target pixel g, and coordinates (x 0 , y 0 ) 
and (x T , y T ) respectively corresponding to the points are 
recognized or calculated in the line segment image memory 5 
(step ST8). The CPU 7 calculates or recognizes coordinates 
of all pixels on the line segment defined by the initial 

25 point and the terminal point in the line segment image 

memory 5 at the subsequent step ST9, and adds, for each of 
the recognized pixels, a level value corresponding to the 
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magnitude E i (x f y) of the level gradient which is calculated 
at the step ST6 to a level value which is currently held by 
the pixel (stored in correspondence with the pixel) and 
stores the result of the addition (step ST10). 
5 The steps STll to ST14 show processing for successively 

shifting a target position (pixel) of an image in the 
inspection area. For each of the pixels at which a level 
gradient has magnitude exceeding a threshold E iLB in the 
inspection area, a line segment along the direction of the 

10 level gradient and having a level corresponding to the 

magnitude of the level gradient is set, to produce a line 
segment image in the same manner as described above. 

A value corresponding to the level of a line segment 
newly set is added to the level value of the pixel which is 

15 recognized to exist on the line segment in the line segment 
image memory 5. Accordingly, the pixel at which line 
segments are overlapped with one another has a higher level 
value than those of the other pixels. The larger the 
magnitude E i (x,y) of the level gradient is, the higher the 

20 level value of a line segment to be set is. Generally, the 
level gradient is large in the contour of a solder ball or a 
void on the image. Accordingly, a lot of line segments 
formed in a portion along the contour of the void are 
overlapped with one another inside the void, so that the 

25 level value of the pixel is significantly high inside the 
void. 

The CPU 7 extracts, when it terminates processing for 
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all the pixels in the inspection area which is a processing 
object, a maximum value of the levels of the pixels in 
the inspection area ( step ST15 ) . The maximum value of 
the detected levels is compared with a predetermined 
5 reference value D LB at the step ST16- When is above D^, 
it is judged that there is a void image in the inspection 
area, and the position of the pixel at which the maximum 
value D max of the levels is obtained is recognized (steps ST17 
and ST19). When is not more than D,^, it is judged that 

10 there is no void image in the inspection area (step ST18). 
When processing for all the inspection areas is 
similarly terminated, the answer is in the affirmative at 
the step ST20. Thereafter, the program proceeds to the step 
ST21. The CPU 7 successively reads out the level values of 

15 the pixels in the line segment image memory 5, to normalize 
each of level values P(x, y) such that it is displaceable by 
an eight bit configuration. 

P(x, y) «- P(x, y) x 255/D^ ... Eg. (10) 

Thereafter, the CPU 7 outputs normalized line segment 

20 image data, together with original input image data and text 
data representing the result of judgment, to the image 
output section 4 to perform synthesis processing, and 
outputs the result of the synthesis to the monitor display 
device 13 (step ST22). The contour of an object may be 

25 extracted and displayed. 

Since at the step ST21, the line segment image data is 
normalized at the maximum luminance level for each of the 
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inspection areas , the maximum luminances after the 
normalization are the same in all the areas. On the other 
hand, when the maximum luminance levels in all the 
inspection areas are detected, and image data in all the 
5 inspection areas are normalized using the maximum luminance 
levels, the luminance is low in the entire inspection area 
where no void exists, so that the inspection area looks 
dark. Accordingly, it is clearly found that there exists no 
void. Image data in all the areas may be, of course, 
10 normalized using a previously determined level, or may not 
be. 

Although in the above-mentioned embodiment, each of the 
line segment images is caused to have a level value 
corresponding to the magnitude of its level gradient, the 

15 present invention is not limited to the same. For example, 
all the line segments may be represented at the same level. 
When a line segment corresponding to a level gradient by 
noises, for example, nonunif ormity of luminance and a line 
segment corresponding to a level gradient in the contour are 

20 formed under the same conditions, however, erroneous 

detection may be performed. Accordingly, it is desirable 
that a level gradient whose magnitude is not more than a 
predetermined threshold as at the step ST7 is excluded from 
a line segment formation processing object. 

25 Figs. 13a and 13b illustrate examples of display of the 

result of judgment which is made by the above-mentioned 
procedure for control. In the example shown in Fig. 13a, 
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only an image 30 in a portion whose level value exceeds the 
threshold D LB is extracted from a line segment image, is so 
converted as to have a high luminance, and is displayed with 
the image superimposed on the original input image. 
5 Character information 31 representing the number of solder 
balls which are judged to be defective due to the presence 
of voids is displayed at an upper left position of a screen. 

In the example of display shown in Fig. 13b, the 
position of an extracted void is reported. A mark 32 
10 representing the position of a pixel at which the maximum 
level value is obtained by the processing at the step 
ST 19 is displayed with the mark superimposed on an input 
image . 

The example of display of the result of inspection is 
15 not limited to the foregoing. A line segment image itself 
may be displayed. Alternatively, the images shown in Figs. 
13a and 13b, an edge image, an image by a pixel satisfying 
E i( x f y) > e ilb' a line segment image, and the like may be 
displayed upon being suitably switched such that the process 
20 of inspection can be confirmed. 

Although in the above-mentioned embodiment, an 
inspection area is set for each solder ball, the present 
invention is not limited to the same. For example, an 
inspection area may be set so as to include all solder balls 
25 on an image. In this case, the same processing as that at 
the steps ST4 to ST10 shown in Fig. 12a is performed once 
with respect to image data in the inspection area, a pixel 
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having a level value exceeding the threshold D M is then 
retrieved, and the position of the pixel is collated with 
the position where each of the solder balls is arranged, 
thereby making it possible to recognize the solder ball 
5 having a void. 

In the above-mentioned embodiment, the luminance level 
of a void which is a detection object is lower than the 
luminance level in its vicinity, so that each of the line 
segments is set toward the inside of the void. When an 

10 object having a luminance level higher than that in its 

vicinity, for example, a solder ball is a detection object, 
however, the line segment must be set outward. 

Furthermore, if a total of three line segments are set 
by not only drawing a line segment in the direction in which 

15 the line segment should be set but also drawing line 

segments in directions at ± 2 degrees from the direction in 
consideration of the effect of an error in gradient 
calculation processing, for example, it is possible to 
further improve the detection precision of an object. 

20 Although in the above-mentioned embodiment, the 

presence or absence of a void inside a solder ball is 
inspected on the basis of an X-ray transmitting image of the 
solder ball, the same method as described above can be 
applied even when the presence or absence of a pinhole on 

25 the surface of a solder is detected using an image obtained 
by picking up a substrate under normal illumination. 

Furthermore, an inspection object is not limited to one 
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in a circular shape. For example, when line segments are 
set by the same method as that in the above-mentioned 
embodiment with respect to an object having a contour shape 
only a part of which is a circular arc, the line segments to 
5 be formed are overlapped with one another in the vicinity of 
the position of the center of the circular arc. In a case 
where a corner of an article is chamfered in a round shape, 
when the above-mentioned processing is performed, letting d 
and D be respectively the minimum value and the maximum 
10 value of a tolerance for a chamfer radius, line segments are 
concentrated only when the chamfer radius is within the 
tolerance. Accordingly, the chamber radius can be simply 
inspected. 

According to the image processing apparatus in the 
15 above-mentioned embodiment, it is also possible to extract 
an object having a rectangular contour shape and an object 
having corners. In this case, line segments directed in two 
directions perpendicular to the direction of the level 
gradient are formed, as shown in Fig. 2b, so that a point of 
20 intersection of the line segments is extracted as the 

position of the corner. When an object in a shape whose 
direction can be specified is taken as a processing object, 
therefore, it is possible to confirm not only the position 
of the object but also the direction in which the object is 
25 directed by the result of the processing. 

Fig. 14 illustrates an example of a gray level image of 
an object having a cross contour shape. The image of the 
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object appears at a higher luminance level than that of an 
image of a background. 

Fig. 15 illustrates the directions of level gradients 
calculated on the contours of the object shown in Fig. 14. 
5 The level gradient is directed perpendicularly to each of 
the sides and toward the background. 

Fig. 16 shows the procedure for inputting the gray 
level image shown in Fig. 14 to the image processing 
apparatus shown in Fig. 5 and measuring the position and the 

10 inclination of an object. Image data obtained by picking up 
the object under normal illumination is fed to the image 
processing apparatus. 

When image data from the camera is inputted at the step 
ST30, the CPU 7 accept designation of an inspection area on 

15 an image, fetches coordinates (x x , y x ) and (x 2 , y 2 ) at upper 
left and lower right vertexes of the designated area, and 
recognizes the coordinates as set positions of the 
inspection area (step ST31), as in the procedure shown in 
Fig. 12a. The CPU 7 clears an area of each of pixels in the 

20 line segment image memory 5 at the subsequent step ST32, 
then successively pays attention to the pixels in the 
inspection area, and performs processing at steps ST34 to 
ST43 for each of the pixels. 

In the above-mentioned loop, the CPU 7 calculates level 

25 gradients dx(x, y) and dy(x, y) in the X-direction and the 
Y-direction of a target pixel g(x, y), as in the procedure 
shown in Figs. 12a and 12b. The direction E c (x, y) of a 
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level gradient and the magnitude E ± (x, y) of the level 
gradient are calculated for each of the pixels (steps ST34 
to ST36). If the calculated magnitude E ± (x, y) of the level 
gradient is larger than a threshold E iLB (step ST37), a first 
5 line segment is set in a direction at an angle of 90 degrees 
in a counterclockwise direction with respect to the detected 
direction E c (x, y) of the level gradient at the step ST38. 
Further, a second line segment is set in a direction at an 
angle of 270 degrees in a counterclockwise direction with 
10 respect to the direction E c (x, y) of the level gradient at 
the step ST39. 

The processing for setting each of the line segments at 
the steps ST38 and ST39 is performed in the same procedure 
as that at the steps ST8 to ST10 shown in Fig. 12a. 

15 When processing of all the pixels in the inspection 

area is completed (steps ST40 to ST43), the CPU 7 checks the 
level value of each of the pixels in the line segment image 
memory 5, and extracts the pixel having a level value which 
is not less than a predetermined threshold D th in the 

20 inspection area (step ST44). 

When with respect to the object on the gray level image 
shown in Fig. 14, line segments are set in two directions 
perpendicular to the direction of its level gradient, a line 
segment images of the line segments which are overlapped 

25 with one another are produced along the length of sides 

constituting the contour of the object. In this case, the 
largest number of lines are concentrated at 12 points of 
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intersection (indicated by • in the drawing) corresponding 
to the corners of the object out of 16 points of 
intersection formed by the line segments in the respective 
directions. Accordingly, the above-mentioned threshold D th 
5 is set between a level value obtained at each of the points 
of intersection and the other points of intersection 
(indicated by x in the drawing), thereby making it possible 
to detect pixels respectively corresponding to the corners 
of the object at the step ST44. 

10 When the pixels respectively corresponding to the 

corners of the object are thus detected, the CPU 7 
calculates the coordinates of a representative point 
representing the position, for example, the center of 
gravity, of the object using the coordinates of each of the 

15 detected positions, specifies principal axes (two axes) of 
the object from the positional relationship between the 
detected positions, and calculates the direction of each of 
the axes in the subsequent step ST45. 

Thereafter, the result of the calculation is outputted 

20 to the monitor display device 13, and the shift in the 
position and the shift in the rotation of the image are 
recognized from the result of the calculation. Accordingly, 
processing corresponding to a purpose, for example, detailed 
recognition processing of the object is performed. 

25 Although all the processing shown in Fig. 12a, the 

processing shown in Fig. 12b, and the processing shown in 
Fig. 16 are performed upon directly accepting an image 
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produced by a camera in the image processing apparatus, the 
present invention is not limited to the same. For example, 
it is also possible to input digital image data produced at 
a distance through the I/O port 14 and process the inputted 
5 digital image data. 

The processing shown in Figs. 12a, Fig. 12b, and Fig. 
16 is performed by the CPU 7 on the basis of a control 
program installed in the hard disk 10. If a part of the 
processing is performed by hardware, however, the speed of 

10 the processing can be further increased by reducing the 
burden on the CPU 7. 

Fig. 18 illustrates another example of the 
configuration of an image processing apparatus, where a 
level gradient measurement section 25 and an overlap 

15 extraction section 2 6 are added in addition to the 
configuration shown in Fig. 5. 

The level gradient measurement section 25 comprises a 
differential processing circuit for successively accepting 
pixel data on an input image stored in the image memory 3 

20 and calculating level gradients dx(x, y) and dy(x, y) in the 
X-direction and the Y-direction and an operation circuit for 
calculating from the level gradients dx(x, y) and dy(x, y) 
the magnitude E i (x, y) of the level gradient and an angle 
E c (x, y) indicating the direction of the level gradient. The 

25 overlap extraction section 26 comprises a circuit 

configuration for successively accepting pixels in the line 
segment image memory 5, extracting the pixel having the 
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maximum value of the luminance level, and comparing the 
maximum value with a threshold D BL or a threshold D th . 

The CPU 7 calculates the magnitude of the level 
gradient and the direction of the level gradient in an 
5 inspection area using the level gradient measurement section 
25, and then sets a line segment in a direction 
corresponding to the shape of an object to be inspected with 
respect to a level gradient whose magnitude is not less than 
a predetermined value, using the result of the calculation. 

10 When the line segment formation processing is completed, the 
overlap extraction section 26 is operated, to retrieve an 
overlap of line segment images, and the CPU 7 performs 
judgment of the presence or absence of the object and 
measurement processing of the position and the inclination 

15 depending on the result of the retrieval. 

The configuration of the image processing apparatus is 
not limited to that in the above-mentioned embodiment. For 
example, the line segment image production processing can be 
also performed by hardware. 
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CLAIMS 

1. An image processing apparatus comprising: 

gradient calculation means for calculating at least the 
direction of the level gradient of a processing unit in 
5 given image data including a plurality of pixels, the pixels 
respectively having level data; 

line segment formation means for producing line segment 
image data representing a line segment having a direction 
corresponding to the direction of the level gradient which 
10 is calculated by said gradient calculation means and a given 
length; and 

line segment image storage means for storing the line 
segment image data produced by said line segment formation 
means . 

15 2. The image processing apparatus according to claim 1, 
further comprising 

image storage means for storing said given image data. 

3. The image processing apparatus according to claim 1 or 

2, further comprising 

20 image data extraction means for extracting image data 

in a processing region set in input image data and feeding 
the extracted image data to said gradient calculation means. 

4. The image processing apparatus according to claim 3, 
further comprising 

25 means for setting said processing region. 

5. The image processing apparatus according to any one of 
claims 1 to 4, wherein 
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said line segment formation means produces line segment 
image data at a gray level. 

6. The image processing apparatus according to any one of 
claims 1 to 4, wherein 

5 said line segment formation means produces line segment 

image data at a binary level. 

7 . The image processing apparatus according to any one of 
claims 1 to 6, wherein 

said gradient calculation means calculates the 
10 magnitude of the level gradient in addition to the direction 
of the level gradient. 

8. The image processing apparatus according to any one of 
claims 1 to 5, wherein 

said gradient calculation means calculates the 
15 magnitude of the level gradient in addition to the direction 
of the level gradient, and 

said line segment formation means produces line segment 
image data having a level corresponding to the magnitude of 
the level gradient which is calculated by said gradient 
20 calculation means. 

9. The image processing apparatus according to any one of 
claims 1 to 6, wherein 

said gradient calculation means calculates the 
magnitude of the level gradient in addition to the direction 
25 of the level gradient, and 

said line segment formation means produces line segment 
image data only when the magnitude of the level gradient 
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which is calculated by said gradient calculation means is 
not less than a predetermined threshold. 

10 . The image processing apparatus according to any one of 
claims 1 to 9, wherein 

5 said line segment image storage means adds new line 

segment image data to line segment image data already stored 
at each of the pixels, and stores the result of the 
addition. 

11. The image processing apparatus according to any one of 
10 claims 1 to 9, wherein 

said line segment image storage means stores new line 
segment image data without subjecting the line segment image 
data to addition processing. 

12 . The image processing apparatus according to any one of 
15 claims 1 to 11, wherein 

said line segment formation means produces a line 
segment having a predetermined length in a direction 
corresponding to the calculated direction of the level 
gradient from the position of the processing unit. 

20 13. The image processing apparatus according to any one of 
claims 1 to 11, wherein 

said line segment formation means produces, when the 
distance from the position of the processing unit to an 
initial point and the distance from the processing unit to a 

25 terminal point are given, a line segment from said initial 

point to said terminal point in a direction corresponding to 
the calculated direction of the level gradient. 
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14. The image processing apparatus according to claim 12, 
further comprising 

means for setting the length of the line segment. 

15. The image processing apparatus according to claim 13, 
5 further comprising 

means for setting at least one of the distance from the 
processing unit to the initial point and the distance from 
the processing unit to the terminal point. 

16. The image processing apparatus according to any one of 
10 claims 1 to 15, further comprising 

means for detecting a portion where line segments 
represented by the line segment image data stored in said 
line segment image storage means are concentrated. 

17. The image processing apparatus according to any one of 
15 claims 1 to 15, further comprising 

means for detecting the position of the pixel having 
the maximum of the levels of the line segment image data 
stored in said line segment image storage means. 

18. The image processing apparatus according to claim 17, 
20 further comprising 

means for judging whether or not said maximum level 
exceeds a predetermined threshold. 

19. The image processing apparatus according to any one of 
claims 1 to 18, further comprising 

25 image input means having a camera for producing image 

data and feeding the produced image data to said gradient 
calculation means . 
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20. The image processing apparatus according to any one of 
claims 1 to 15 and claim 19 , further comprising 

a display device for displaying a line segment image 
represented by the line segment image data produced by said 
5 line segment formation means or the line segment image data 
stored in said line segment image storage means . 

21. The image processing apparatus according to claim 20, 
wherein 

said display device displays an image represented by 
10 said given image data with the image overlapped with said 
line segment image. 

22. The image processing apparatus according to claim 20, 
further comprising 

means for extracting an edge of the image represented 
15 by said given image data, 

said display device displaying an image represented by 
the extracted edge with the image overlapped with said line 
segment image. 

23. The image processing apparatus according to claim 16, 
20 further comprising 

a display device for displaying a mark representing the 
portion, where the line segments are concentrated, detected 
by said detection means with the mark overlapped with the 
image represented by said image data. 
25 24. The image processing apparatus according to claim 17, 
further comprising 

a display device for displaying, at the position of the 
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pixel having the maximum level which is detected by said 
detection means, a mark indicating that the pixel has the 
maximum level with the mark overlapped with the image 
represented by said image data. 

25. The image processing apparatus according to claim 18, 
further comprising 

a display device for displaying, at the position of a 
pixel having the maximum level which is judged to exceed a 
threshold by said judgment means, a mark indicating that the 
pixel has the maximum level with the mark overlapped with 
the image represented by said image data. 

26. The image processing apparatus according to any one of 
claims 23 to 25, further comprising 

means for extracting an edge of the image represented 
by said image data, 

said display device displaying an image represented by 
the edge extracted by said edge extraction means in addition 
to or in place of the image represented by said image data. 

27. The image processing apparatus according to claim 26, 
wherein 

said display device displays the line segment image 
represented by said line segment image data with the line 
segment image further overlapped with the image represented 
by the edge. 

28. An image processing apparatus comprising: 

image processing means for calculating at least the 
direction of the level gradient of a processing unit in 
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given image data, and producing line segment image data 
representing a line segment having a direction corresponding 
to the calculated direction of the level gradient; and 
display means for displaying a line segment image 
5 represented by the line segment image data produced by said 
image processing means. 

29. The image processing apparatus according to claim 28, 
wherein 

said display device displays the image represented by 
10 said image data with the image overlapped with said line 
segment image. 

30. The image processing apparatus according to claim 29, 
further comprising 

means for extracting an edge of the image represented 
15 by said image data, 

said display device displaying an image represented by 
the edge extracted by said edge extraction means in addition 
to or in place of the image represented by said image data. 

31. An image processing method comprising the steps of: 
20 calculating at least the direction of the level 

gradient of a processing unit in given image data including 
a plurality of pixels, the pixels respectively having level 
data; 

producing line segment image data representing a line 
25 segment having a direction corresponding to the calculated 
direction of the level gradient and a given length; and 

storing the produced line segment image data in storage 



58 



means . 

32. A medium storing a program for controlling a computer 
so as to: 

calculate at least the direction of the level gradient 
5 of a processing unit in given image data including a 

plurality of pixels, the pixels respectively having level 
data; 

produce line segment image data representing a line 
segment having a direction corresponding to the calculated 
10 direction of the level gradient and a given length; and 

store the produced line segment image data in storage 

means . 

33. An image processing method comprising: 

calculating at least the direction of the level 
15 gradient of a processing unit in given image data, and 
producing line segment image data representing a line 
segment having a direction corresponding to the calculated 
direction of the level gradient; and 

displaying a line segment image represented by the 
20 produced line segment image data on a display device. 

34. A medium storing a program for controlling a computer 
so as to: 

calculate at least \he direction of the level gradient 
of a processing unit in given image data, and produce line 
25 segment image data representing a line segment having a 

direction corresponding to the calculated direction of the 
level gradient; and 
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display a line segment image represented by the 
produced line segment image data on a display device. 

35. An image processing apparatus comprising: 

means for extracting a plurality of edges whose level 
5 gradients are not less than a predetermined value in given 
image data; 

means for setting, for each of the edges, a line 
segment extending in a direction corresponding to the 
direction of the extracted edge; and 
10 means for detecting the presence or absence of a point 

of intersection of a plurality of line segments and the 
position thereof. 

36. The image processing apparatus according to claim 35, 
wherein 

15 the direction of the line segment is a direction 

perpendicular to the direction of the edge or the same 
direction as the direction of the edge. 

37. An inspection apparatus comprising: 

image input means for inputting image data representing 
20 an inspection object; 

means for calculating at least the direction of the 
level gradient of a processing unit in said input image 
data, and producing line segment image data representing a 
line segment having a direction corresponding to the 
25 calculated direction of the level gradient; and 

means for detecting the presence or absence of a 
portion where line segment images are concentrated or are 
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overlapped with one another and the position thereof on the 
basis of the produced line segment image data. 

38. The inspection apparatus according to claim 37, wherein 

the direction corresponding to the direction of said 
5 level gradient is the direction of the level gradient or a 
direction perpendicular to the direction of the level 
gradient . 

39. The inspection apparatus according to claim 38, further 
comprising 

10 a display device for displaying the line segment image 

on the basis of said line segment image data. 

40. The inspection apparatus according to claim 39, wherein 

said display device displays an image of the object 
represented by said input image data with the image 
15 overlapped with the line segment image. 

41. The inspection apparatus according to claim 37 or 38, 
further comprising 

a display device for displaying the detected position 
of the portion where the line segment images are 
20 concentrated or are overlapped with one another on the image 
of the object represented by said input image data or an 
image represented by an edge extracted from said input image 
data. 

42. The inspection apparatus according to any one of claims 
25 37 to 41, further comprising 

means for inputting data relating to the length of the 
line segment or the initial point and the terminal point of 
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the line segment. 



ABSTRACT OF THE DISCLOSURE 
In a gray level image, the direction and the magnitude 
of a level gradient are found for each of pixels. With 
respect to the pixel having a level gradient whose magnitude 
5 exceeds a predetermined value, a line segment having a 

predetermined length is drawn in the direction of the level 
gradient from the pixel or a position spaced apart from the 
pixel by a predetermined distance. The luminance level of 
the line segment corresponds to the magnitude of the level 
10 gradient. A portion where a lot of line segments are 

overlapped with one another or a portion where the line 
segments are concentrated is detected, to recognize the 
center, the corners, etc. of an object appearing on the 
image . 
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Atty. Dkt. #: 



DECLARATION AND POWER OF ATTORNEY 
FOR UNITED STATES PATENT APPLICATION 



As a below named inventor, I hereby declare that: 



My residence, post office address and citizenship are as stated below next to my name; and 

I believe I am the original, first and sole inventor (if only one name is listed below) or an original, first and joint 
inventor (if plural names are listed below) of the subject matter which is claimed and for which a patent is sought 
on the invention entitled IMAGE PROCESSING APPARATUS AND METHOD, MEDIUM STORING PROGRAM 
FOR IMAGE PROCESSING. AND INSPECTION APPARATUS . 
the specification of which 
(check one) | x [ is attached hereto. 

| | was filed on as 

Application Serial No. 

and was amended on (if applicable) 

I hereby state that I have reviewed and understand the contents of the above-identified specification, including 
the claims, as amended by any amendment referred to above. 

I acknowledge the duty to disclose to the U.S. Patent and Trademark Office all information which is known to 
me to be material to patentability to the examination of this application in accordance with Title 37, Code of 
Federal Regulations, §1.56 and Title 35, United States Code, §102. 

I hereby claim foreign priority benefit under Title 35, United States Code, §1 19 of any foreign application^) for 
patent or inventor's certificate listed below and have also identified below any foreign application for patent or 
inventor's certificate having a filing date before that of the application on which priority is claimed: 

Prior Foreign Application^) 

Priority Claimed 



(Country) 



(Number) 



(Country) 



(Number) 



(Country) 



07/04 /I 998 
(Day/Month/Year Filed) 



(Day/Month/Year Filed) 



(Day/Month/Year Filed) 



[x^Yes No 
QYes No 
Q]Yes No 



| [ Additional Prior Foreign Applications are being listed on separate sheet(s) attached hereto. 
I hereby claim the benefit under Title 35, United States Code, §120 of any United States application^) listed 
below and, insofar as the subject matter of any of the claims of this application is not disclosed in the prior 
United States application in the manner provided by the first paragraph of Title 35, United States Code, §112, 1 
acknowledge the duty to disclose to the U.S. Patent and Trademark Office all information known to me to be 
material to patentability as defined in Title 37, Code of Federal Regulations, §1.56 which became available 
between the filing date of the prior application and the national or PCT international filing date of this 
application: 
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Application Serial No. Filing Date Status - patented, pending, abandoned 



Application Serial No. Filing Date Status - patented, pending, abandoned 



Application Serial No. Filing Date Status - patented, pending, abandoned 

| | Additional United States Applications are being listed on separate sheet(s) attached hereto. 
As a named inventor, I hereby appoint: 

Gary M. Hoffman, Reg. No. 26,411 ; Thomas J. D'Amico, Reg. No. 28,373 ; Donald A. Gregory, Reg. No. 
_28-354f James W. Brady, Jr., Reg. No. 32*115; Jon D. Grossman, Reg. No. 32,699 : Mark J. Thronson, Reg. 

No. 3^a82;,Eric Oliver, Reg. No. 3 5.307; John A. Wasleff, Reg. No..^fijM7; Laurence E. Fisher, Reg. No. 
.37-434^ John Fuisz, Reg. No. 3X329_U_Robert L. Hails, Jr., Reg. No. 39.702: William E. Powell, III, Reg. No. 
J39JJ03j_and James M. Silbermann, Reg. No. 40Jd2u^ 

my attorneys with full power of substitution and revocation to prosecute this application and to receive 
correspondence from and transact all business in the Patent and Trademark Office connected therewith. 

Address all correspondence to: 

DICKSTEIN SHAPIRO, MORIN & OSHINSKY LLP 
2101 L Street NW ~ 
Washington, DC 20037 
(202) 785-9700 

The undersigned hereby authorizes the U.S. attorneys and/or agents named herein to accept and follow 
instructions from the agents and/or liaisons of the undersigned and/or the Assignee of this application as to any 
action to be taken in the Patent and Trademark Office regarding this application without direct communication 
between the U.S. attorney or agent and the undersigned. In the event of a change in the persons from whom 
instructions may be taken, the U.S. attorneys and/or agents named herein will be so notified by the 
undersigned and/or any Assignee of this application. 

I hereby declare that all statements made herein of my own knowledge are true and that ail statements made 
on information and belief are believed to be true; and further that these statements were made with the 
knowledge that willful false statements and the like so made are punishable by fine or imprisonment, or both, 
under Section 1 001 of Title 1 8 of the United States Code and that such willful false statements may jeopardize 
the validity of the application or any patent issued thereon. 



Full name of sole or first inventor: ..shiro FUJIEDA. ... - -- - _ &&>}>eV'20J? ?9' 

Inventor's signature: jSt ^^u^j^J >ldi^ ^J/eJev Date: £ d&hi}S» )$$$S.P 
Residence: Kyoto-*shi, Kyoto. JapanO Citizenship: Japanese 
Post Office Address: c/o OMRON Corporation 

10. Tsuchido-cho. Hanazono, Ukvo-^ku. Kyoto^shi. 

Kyoto 616-^8025, Japan 
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Full name of second inventor: 

Inventor's signature: Date: 

Residence: Citizenship: 

Post Office Address: 



Full name of third inventor: 

Inventor's signature: Date: 

Residence: Citizenship: 

Post Office Address: 



| | Additional inventors are being named on separate sheet(s) attached hereto. 

Title 37, Code of Federal Regulations, § 1.56 Duty to disclose information material to 

patentability 

(a) A patent by its very nature is affected with a public interest. The public interest is best served, and the 
most effective patent examination occurs when, at the time an application is being examined, the Office is 
aware of and evaluates the teachings of all information material to patentability. Each individual associated 
with the filing and prosecution of a patent application has a duty of candor and good faith in dealing with the 
Office, which includes a duty to disclose to the Office al! information known to that individual to be material to 
patentability as defined in this section. The duty to disclose information exists with respect to each pending 
claim until the claim is cancelled or withdrawn from consideration, or the application becomes abandoned. 
Information material to the patentability of a claim that is cancelled or withdrawn from consideration need not 
be submitted if the information is not material to the patentability of any claim remaining under consideration in 
the application. There is no duty to submit information which is not material to the patentability of any existing 
claim. The duty to disclose all information known to be material to patentability is deemed to be satisfied if all 
information known to be material to patentability of any claim issued in a patent was cited by the Office or 
submitted to the Office in the manner prescribed by §§1 .97(b)-(d) and 1 .98. However, no patent will be granted 
on an application in connection with which fraud on the Office was practiced or attempted or the duty of 
disclosure was violated through bad faith or intentional misconduct. The Office encourages applicants to 
carefully examine: (1) prior art cited in search reports of a foreign patent office in a counterpart application, 
and (2) the closest information over which individuals associated with the filing or prosecution of a patent 
application believe any pending claim patentability defines, to make sure that any material information 
contained therein is disclosed to the Office. 

Title 35, United States Code, § 1 02 Conditions for patentability; 

novelty and loss of right to patent 

A person shall be entitled to a patent unless - 

(a) the invention was known or used by others in this country, or patented or described in a printed 
publication in this or a foreign country, before the invention thereof by the applicant for patent, or 

(b) the invention was patented or described in -a printed- publication in this or a-foreign country or in public 
use or on sale in this country, more than one year prior to the date of the application for patent in the United 
States, or 

(c) he has abandoned the invention, or 

(d) the invention was first patented or caused to be patented, or was the subject of an inventor's 
certificate, by the applicant or his legal representatives or assigns in a foreign country prior to the date of the 
application for patent in this country on an application for patent or inventor's certificate filed more than twelve 
months before the filing of the application in the United States, or 
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(e) the invention was described in a patent granted on an application for patent by another filed in the 
United States before the invention thereof by the applicant for patent, or on an international application by 
another who has fulfilled the requirements of paragraphs (1), (2), and (4) of section 371(c) of this title before 
the invention thereof by the applicant for patent, or 

(f) he did not himself invent the subject matter sought to be patented, or 

(g) before the applicant's invention thereof the invention was made in this country by another who had not 
abandoned, suppressed, or concealed it. In determining priority of invention there shall be considered not only 
the respective dates of conception and reduction to practice of the invention, but also the reasonable diligence 
of one who was first to conceive and last to reduce to practice, from a time prior to conception by the other. 

Title 35, United States Code, § 1 03 Conditions for patentability; 

non-obvious subject matter 

A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 1 02 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a 
person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived 
by the manner in which the invention was made. 

Subject matter developed by another person, which qualifies as prior art only under subsection (f) or (g) of 
section 102 of this title, shall not preclude patentability under this section where the subject matter and the 
claimed invention were, at the time the invention was made, owned by the same person or subject to an 
obligation of assignment to the same person. 

Title 35, United States Code, § 112 Specification 

The specification shall contain a written description of the invention, and of the manner and process of 
making and using it, in such full, clear, concise and exact terms as to enable any person skilled in the art to 
which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the 
best mode contemplated by the inventor of carrying out his invention. ... 

Title 35, United States Code, § 119 Benefit of earlier filing date in 

foreign country; right of priority 

An application for patent for an invention filed in this country by any person who has, or whose legal 
representatives or assigns have, previously regularly filed an application for a patent for the same invention in 
a foreign country which affords similar privileges in the case of applications filed in the United States or to 
citizens of the United States, shall have the same effect as the same application would have if filed in this 
country on the date on which the application for patent for the same invention was first filed in such foreign 
country, if the application in this country is filed within twelve months from the earliest date on which such 
foreign application was filed; but no patent shall be granted on any application for patent for an invention which 
had been patented or described in a printed publication in any country more than one year before the date of 
the actual filing of the application in this country, or which had been in public use or on sale in this country 
more than one year prior to such filing. 

No application for patent shall be entitled to this right of priority unless a claim therefor and a certified 
copy of the original foreign application, specification and drawings upon which it is based are filed in the Patent 
and Trademark Office before the patent is granted, or at such time during the pendency of the application as 
required by the Commissioner not earlier than six months after the filing of the application in this country. Such 
certification shall be made-by the patent -office of the- foreign country-in- which -filed -and show the date of the 
application and of the filing of the specification and other papers. The Commissioner may require a translation 
of the papers filed if not in the English language and such other information as he deems necessary. 

In like manner and subject to the same conditions and requirements, the right provided in this section may 
be based upon a subsequent regularly filed application in the same foreign country instead of the first filed 
foreign application, provided that any foreign application filed prior to such subsequent application has been 
withdrawn, abandoned, or otherwise disposed of, without having been laid open to public inspection and without 
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leaving any rights outstanding, and has not served, nor thereafter shall serve, as a basis for claiming a right of 
priority. 

Applications for inventor's certificates filed in a foreign country in which applicants have a right to apply, at 
their discretion, either for a patent or for an inventor's certificate shall be treated in this country in the same 
manner and have the same effect for purpose of the right of priority under this section as applications for 
patents, subject to the same conditions and requirements of this section as apply to applications for patents, 
provided such applicants are entitled to the benefit of the Stockholm Revision of the Paris Convention at the 
time of such filing. 
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